Skip to content

Fixes #16018: Existential widening for wildcard arguments#26152

Open
He-Pin wants to merge 2 commits into
scala:mainfrom
He-Pin:add-i16018-tests
Open

Fixes #16018: Existential widening for wildcard arguments#26152
He-Pin wants to merge 2 commits into
scala:mainfrom
He-Pin:add-i16018-tests

Conversation

@He-Pin
Copy link
Copy Markdown
Contributor

@He-Pin He-Pin commented May 23, 2026

Fixes #16018

A long-standing wildcard-subtyping gap in TypeComparer.compareCaptured was rejecting F[? <: hi] <:< F[hi] for covariant F (dually for contravariant G), even though existential elimination over a variant type constructor admits exactly that widening. The fix consults the wildcard's own bound, intersected with the parameter's declared bound, instead of the declared bound alone — purely additive, mirrors the same lattice operation Scala 2 implements, and closes the original akka-derived ticket along with the broader pattern.

How much have you relied on LLM-based tools in this contribution?

Extensively, for: minimizing the akka reproducer, instrumenting TypeComparer to localize the failing subtype path, identifying the wrong-field consultation (paramBounds(tparam).hi vs arg1.hi), proving the fix is type-theoretically correct, and designing the directional pos / neg matrix. I have personally reviewed every line of the diff, can explain each step, and have stress-tested the fix against the full local CompilationTests suite plus targeted wildcard / match / variance filter subsets before submitting.

How was the solution tested?

New automated tests (including the issue's reproducer).

Run via sbt 'scala3-compiler-bootstrapped/testOnly dotty.tools.dotc.CompilationTests':

  • i16018 filter — all five test files (16/16 pass).
  • wildcard / match / variance filter subsets — 16/16 each.
  • Full CompilationTests run — the residual failures (tests/run/i13358.scala, tests/run/lazy-*.scala, tests/run/t5552.scala, tests/run/t7406.scala, tests/run/isInstanceOf-eval.scala, tests/run/i24553.scala, tests/pos-custom-args/captures/fill-cbn.scala, capture-checking neg suite) all reproduce on main without this change and are caused by JDK 25 environment drift (sun.misc.Unsafe deprecation messages on stdout, the introduction of java.lang.IO, removal of native from Object.wait). Verified by re-running them on a clean main checkout.

Diagnosis — how the bug was located

The ticket and its successive minimizations all reduced to the same symptom: a collect { case g: Sub[…] @unchecked => g; case other => other } over a value derived from a java.lang.Iterable[? <: G[? <: M]] rejects the second case with

Found:    (other : ?N.CAP)
Required: Sub[…] & ?N.CAP
where:    ?N is an unknown value of type scala.runtime.TypeBox[Nothing, G[? <: M]]

Instrumentation of TypeComparer.compareTypeParamRef and addOneBound exposed two layered issues:

Layer 1 — sequential constraint accumulation on the partial-function result variable B. The first case body's pattern-narrowed type Sub[…] ∩ ?N.CAP was added as B's lower bound, after which the second case body's check ?N.CAP <:< B was rejected because ?N.CAP ⊄ Sub[…] ∩ ?N.CAP. The fix point we actually want is B := ⨆ᵢ T_i (the LUB of case body types), namely ?N.CAP.

Layer 2 — and this turned out to be the root cause — even bypassing layer 1, the surrounding adaptation reduced to F[? <: M] <:< F[M] for covariant F (after Seq's covariance). That check went through TypeComparer.compareCaptured and was rejected. A reduced standalone reproduction confirmed the regression independently of pattern matching:

class Box[+M]
def f[M](xs: Seq[Box[? <: M]]): Seq[Box[M]] = xs    // also rejected pre-fix

So the partial-function symptom was downstream of a more fundamental gap in wildcard subtyping under variance. Fixing layer 2 dissolves layer 1, and is what this PR does.

Root cause

TypeComparer.compareCaptured is the fallback for an applied-type argument check when the source argument is a wildcard arg1: TypeBounds and the target argument is a concrete arg2. Pre-fix it asked

v > 0:  isSubType(paramBounds(tparam).hi, arg2)
v < 0:  isSubType(arg2, paramBounds(tparam).lo)

— i.e. it consulted the declared parameter bound, which for an unconstrained T collapses to Any (covariant case) or Nothing (contravariant case), so the conformance check became Any <: arg2 and almost always failed. The wildcard's own bound (arg1.hi / arg1.lo) — which is the only place where the user's intent ? <: hi was recorded — was never consulted.

Type-theoretic justification

Order subtypes (<:) so that larger means more general. For each variance direction, the existential F[? >: lo <: hi] denotes a family of types parameterised by the witness X, and we ask for the least upper bound () of that family — the smallest type into which every member of the family can be widened. The fact that a single concrete type is the LUB is what makes the widening rule sound (admissibility of existential elimination at a variant position).

Covariant F[+T]. The family is {F[X] | X <: hi}. By covariance,

X₁ <: X₂  ⇒  F[X₁] <: F[X₂]   (functoriality preserves the order)

so the family is order-preserving in X; its LUB is attained at X = hi:

⨆ { F[X] | X <: hi }  =  F[hi]        ⇒   F[? <: hi]  <:  F[hi]

Contravariant G[-T]. The family is {G[X] | X >: lo}. By contravariance,

X₁ <: X₂  ⇒  G[X₂] <: G[X₁]   (functoriality reverses the order)

so the family is order-reversing in X; its LUB is attained at X = lo (note: the supremum sits at the small end of the witness, because the type constructor flips the order):

⨆ { G[X] | X >: lo }  =  G[lo]        ⇒   G[? >: lo]  <:  G[lo]

Both rules use the same lattice operation (LUB) and the same notion of "widening"; the only thing that differs is which witness attains the supremum, because the variance of F / G decides whether the witness ordering and the result ordering agree or disagree.

Invariant positions admit neither widening (the family {F[X] | X <: hi} for invariant F has no nontrivial supremum in the family itself), and mixed variance / bound combinations (covariant + lower, contravariant + upper) similarly have no usable LUB — those cases must remain rejected. This is what tests/neg/i16018c.scala pins.

Scala 2 implements both rules; Scala 3 had regressed.

Fix

In TypeComparer.compareCaptured, consult the wildcard's own bound, intersected (resp. unioned) with the parameter's declared bound. The intersection is the declared-bound guard (a completeness guard, not a soundness one — without it NAIVE = arg1.hi would produce false negatives like rejecting CoBounded[? <: Any] <:< CoBounded[Holder]): if a wildcard is wider than what the parameter permits (e.g. F[T <: Holder] applied to ? <: Any), the effective bound is the tighter of the two.

v > 0:  isSubType(arg1.hi & paramBounds(tparam).hi, arg2)
v < 0:  isSubType(arg2, arg1.lo | paramBounds(tparam).lo)

This is purely additive: it never accepts a subtyping that was already accepted by a different path; it only admits the cases the LUB rule above justifies. Three implementations are distinguishable on the test matrix below:

OLD     : isSubType( paramBounds(tparam).hi,                 arg2 )
NAIVE   : isSubType( arg1.hi,                                arg2 )
THE FIX : isSubType( arg1.hi & paramBounds(tparam).hi,       arg2 )

OLD ignores the wildcard's bound: for unbounded Co[+T] fed ? <: M it asks Any <: M and rejects a valid widening. NAIVE ignores the parameter's declared bound: for CoBounded[+T <: Holder] fed ? <: Any it asks Any <: Holder and rejects another valid widening. All three are sound (none accepts a subtyping that violates the bounds either party records); the distinction is completeness. THE FIX is the intersection — the only one of the three that is also complete on this fragment, admitting every existential elimination justified by variance + bounds.

Tests (directional matrix)

file role
tests/pos/i16018.scala mbovel's minimization + pure-Scala matrix with subtype patterns + fallback case other
tests/pos/i16018b.scala akka-style shapes: java.lang.Iterable[? <: G[? <: M]] / java.util.List[…] routed through seqOf, then collect / map
tests/pos/i16018-orig.scala the verbatim akka-minimized reproduction from the ticket (only _? syntax migration), so the original ticket cannot regress
tests/pos/i16018c.scala positive matrix for the fix itself. §1–3 distinguish OLD from THE FIX (the covariant headline and its dual); §4–5 distinguish a hypothetical NAIVE refactor (drop the intersection / union) from THE FIX — they pin the declared-bound guard (NAIVE = arg1.hi alone produces false negatives like rejecting CoBounded[? <: Any] <:< CoBounded[Holder]; THE FIX accepts it) so future code archaeology cannot quietly delete the & paramBounds / | paramBounds
tests/neg/i16018c.scala negative matrix: invariant containers, covariant + lower, contravariant + upper, and boundary checks for the declared-bound guard — CoBounded[? <: Any] must conform at CoBounded[Holder] (pinned positively) but not at CoBounded[Sub] for Sub <: Holder (a strict subtype is out of range, not unsound); dual for the contravariant lo case. These boundary checks pin the upper / lower stop of the widening (all three implementations reject them), not the distinguishing behaviour between OLD / NAIVE / THE FIX

Any future refactor of compareCaptured will turn one of the two halves red the moment any of the three implementations above stops behaving as required.

@He-Pin He-Pin marked this pull request as draft May 23, 2026 16:05
@He-Pin He-Pin force-pushed the add-i16018-tests branch from 98b1671 to b45a8f5 Compare May 23, 2026 17:21
@He-Pin He-Pin force-pushed the add-i16018-tests branch from b45a8f5 to 5ccc481 Compare May 23, 2026 20:27
@He-Pin He-Pin changed the title Pin partial progress of #16018 with directional pos/neg tests Fixes #16018: Existential widening for wildcard arguments May 23, 2026
@He-Pin
Copy link
Copy Markdown
Contributor Author

He-Pin commented May 23, 2026

This fix will facilitate cross-compilation between Scala 2.13 and Scala 3, especially for Java users.

@He-Pin He-Pin force-pushed the add-i16018-tests branch from 5ccc481 to ca8701f Compare May 23, 2026 20:51
Comment thread tests/pos/i16018c.scala Outdated
@He-Pin He-Pin force-pushed the add-i16018-tests branch from ca8701f to d43b013 Compare May 23, 2026 21:14
Comment thread tests/neg/i16018c.scala Outdated
@He-Pin He-Pin force-pushed the add-i16018-tests branch from d43b013 to dc0d66b Compare May 23, 2026 21:34
@He-Pin He-Pin marked this pull request as ready for review May 23, 2026 21:38
@He-Pin He-Pin force-pushed the add-i16018-tests branch from dc0d66b to b952dfd Compare May 23, 2026 21:42
@He-Pin He-Pin marked this pull request as draft May 23, 2026 23:25
@He-Pin He-Pin marked this pull request as ready for review May 23, 2026 23:46
@He-Pin He-Pin marked this pull request as draft May 25, 2026 06:10
@He-Pin He-Pin marked this pull request as ready for review May 25, 2026 16:42
He-Pin added 2 commits May 28, 2026 13:40
The Scala 3 typer was rejecting subtypings of the form

    F[? >: lo <: hi]  <:  F[X]

for covariant or contravariant `F`, even when `X = hi` (covariant) or
`X = lo` (contravariant). This is a regression against Scala 2 and surfaced
as the akka-derived report in scala#16018:

    def f[M](xs: java.lang.Iterable[? <: Container[Any, ? <: M]])
        : Seq[Container[Any, M]] =
      seqOf(xs).collect {
        case g: SubContainer[Any, M] @unchecked => g
        case other                              => other
      }

Once `Inferencing.captureWildcards` lifted the Java wildcard to a
`TypeBox.CAP` skolem and pattern matching narrowed the first case body
to `pat ∩ ?N.CAP`, the inferred result reduced to a chain of subtype
checks that ultimately required `F[? <: M] <:< F[M]` for covariant `F`.
That check went through `TypeComparer.compareCaptured` and was answered
in the negative.

Type-theoretic justification
----------------------------
For a covariant type constructor `F` and an existential `∃X. X <: hi`,

    ⨆{F[X] | X <: hi} = F[hi]                          (covariant supremum)

— the supremum is attained at `X = hi`, by covariance. The compiler must
admit that supremum as a subtype of `F[hi]`. Dually for contravariance with
the lower bound:

    ⨅{G[X] | X >: lo} = G[lo]                          (contravariant infimum)

This is the standard existential-elimination rule for variant occurrences.

Bug
---
`compareCaptured` was checking

    v > 0:  isSubType(paramBounds(tparam).hi, arg2)
    v < 0:  isSubType(arg2, paramBounds(tparam).lo)

i.e. it asked whether the *declared parameter bound* (which collapses to
`Any` / `Nothing` for unconstrained type parameters) conforms to `arg2`,
instead of asking whether the *wildcard's own bound* (`arg1.hi` / `arg1.lo`)
does. That rejected every interesting case.

Fix
---
In `TypeComparer.compareCaptured` use the wildcard's own hi/lo, intersected
(resp. unioned) with the declared parameter bounds to preserve soundness in
the corner case where a wildcard is wider than its parameter permits:

    v > 0:  isSubType(arg1.hi & paramBounds(tparam).hi, arg2)
    v < 0:  isSubType(arg2, arg1.lo | paramBounds(tparam).lo)

Tests (directional matrix)
--------------------------
- `tests/pos/i16018.scala`  — mbovel's minimization plus a pure-Scala matrix
  that exercises subtype patterns + fallback `case other` over wildcards.
- `tests/pos/i16018b.scala` — the akka-style shapes that motivated the
  ticket: `java.lang.Iterable[? <: G[? <: M]]` / `java.util.List[…]`
  routed through `seqOf`, then `collect` / `map` with subtype + fallback
  patterns.
- `tests/pos/i16018c.scala` — positive half of the variance × bound matrix
  for the fix itself: covariant + upper, contravariant + lower, nested,
  via method type params, with parameter-bound tightening.
- `tests/neg/i16018c.scala` — negative half of the matrix: invariant
  containers, covariant + lower, contravariant + upper, and the soundness
  guard against wildcards wider than the required type. These must remain
  rejected after the fix.

Verified locally
----------------
- All four `i16018*` tests: 16/16 pass via
  `sbt 'scala3-compiler-bootstrapped/testOnly … CompilationTests'`
  with `-Ddotty.tests.filter=i16018`.
- `wildcard` / `match` / `variance` filter subsets: 16/16 each.
- Full `CompilationTests` run: the remaining 4 failures
  (`tests/run/i13358.scala`, `tests/run/lazy-*.scala`,
  `tests/run/t5552.scala`, `tests/run/t7406.scala`,
  `tests/run/isInstanceOf-eval.scala`,
  `tests/run/i24553.scala`,
  `tests/pos-custom-args/captures/fill-cbn.scala`, and the
  capture-checking neg suite) reproduce on `main` without this change and
  are caused by JDK 25 environment drift (`sun.misc.Unsafe` deprecation
  messages on stdout, the introduction of `java.lang.IO`, and removal of
  `native` from `Object.wait`).
…ounds

Motivation
----------
The widening fix in b952dfd admits subtypings for F-bounded type
constructors such as `class C[+T <: C[T]]` at the surface, but the
resulting typed tree fails `-Ycheck:all` under frozen constraints with
assertions of the form `M <:< xs.T` / `C[M] <:< C[xs.T]`. Without a
guard, this surface-pass / Ycheck-fail combination silently produces an
internally inconsistent typed tree (compile reports 0 errors).

Modification
------------
In `TypeComparer.isSubArgs`, precompute `recursiveParamBounds`: the set
of `tparams2` whose declared bounds recursively refer to themselves. The
detection visits `tparam.info` via `TypeAccumulator`:

  - direct `TypeRef` to the parameter        -> recursive
  - any `LazyRef`                            -> recursive (conservative,
                                                avoids forcing recursive
                                                class-header init)

`compareCaptured` consults `hasRecursiveParamBounds(tparam)` *before*
the stable capture conversion branch, so both stable capture and
existential widening are short-circuited for recursive bounds, falling
back to the pre-fix conservative `false`. The set is cached via
`lazy val` to amortise across arg positions in the same `isSubArgs`
invocation.

Tests
-----
- `tests/neg/i16018d.scala` -- F-bound guard single-point pin
  (`class FBounded[+T <: FBounded[T]]`). Guard necessity was verified
  out-of-band via a temporary `if (false && hasRecursiveParamBounds...)`:
  with the guard disabled, this test compiles with 0 errors.
- `tests/pos/i16018d.scala` -- four cases not covered by the original
  directional matrix: Function1 nested under Co, intersection upper
  bound, path-dependent upper bound, and acyclic dependent bound
  (reverse-pin proving the guard does not over-restrict to all
  dependent bounds, only ones with a self-reference).
- `tests/neg/i16018e.scala` -- three HKT-nested variance cases
  (Function1 contra+upper, Function1 co+lower, Inv-in-Co) pinning
  per-position variance checks through HKT nesting.

Verified locally
----------------
`testOnly dotty.tools.dotc.CompilationTests
   -- -Ddotty.tests.filter=i16018` -> 16 passed, 0 failed.
@He-Pin He-Pin force-pushed the add-i16018-tests branch from 1f58a30 to 1605f0f Compare May 28, 2026 05:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Type computed to any, works in Scala 2

1 participant